This invention relates generally to the field of the multimedia applications. More particularly, this invention relates to a new frame type, apparatus and method for using same to provide for access of a video stream.
Multimedia applications that include audio and streaming video information have come into greater use. Several multimedia groups have established and proposed standards for compressing/encoding and decompressing/decoding the audio and video information. The examples are MPEG standards, established by the Motion Picture Expert Group and standards developed by ITU-Telecommunications Standardization.
The following are incorporated herein by reference:
G. Bjontegaard, xe2x80x9cH.26L Test Model Long Term Number 6 (TML-6) draft0xe2x80x9d, document VCEG-L45, ITU-T Video Coding Experts Group Meeting, Eibsee, Germany, Jan. 9-12, 2001. Keiichi Hibi, xe2x80x9cReport of the Ad Hoc Committee on H.26L Developmentxe2x80x9d, document Q15-H-07, ITU-T Video Coding Experts Group (Question 15) Meeting, Berlin, Aug. 3-6, 1999. Gary S. Greenbaum, xe2x80x9cRemarks on the H.26L Project: Streaming Video Requirements for Next Generation Video Compression Standardsxe2x80x9d, document Q15-G-11, ITU-T Video Coding Experts Group (Question 15) Meeting, Monterey, Feb. 16-19, 1999. G. Bjontegaard, xe2x80x9cRecommended Simulation Conditions for H.26Lxe2x80x9d, document Q15-I-62, ITU-T Video Coding Experts Group (Question 15) Meeting, Red Bank, N.J., Oct. 19-22, 1999. ATM and MPEG-2 Integrating Digital Video into Broadband Networks by Michael Orzessek and Peter Sommer (Prentice Hall Upper Saddle River N.J.).
Video sequences, like ordinary motion pictures recorded on film, comprise a sequence of still images, and the illusion of motion is created by displaying consecutive images at a relatively fast rate. For example, the display rate are between five and thirty frames per second. Because of the relatively fast frame rate, the images in consecutive frames tend to be similar. A typical scene recorded by a camera comprises some stationary elements, such as, for example, background scenery and some moving parts. The moving parts may take many different forms, for example, the face of a news reader, moving traffic, and so on. Alternatively, the camera recording the scene may itself be moving, in which case all elements of the image have the same kind of motion. In many cases, this means that the overall change between one video frame and the next is rather small. Of course, this depends on the nature of the movement, the rate of the movement, i.e., the amount of change from one frame to the next.
The purpose of the video coding is to remove the redundancy in the image sequence so that the encoded data rate is commensurate with the available bandwidth to transport the video sequence while keeping the distortion between the original and reconstructed images as small as possible. The redundancy in video sequences can be categorized into spatial and temporal redundancy. Spatial redundancy refers to the correlation between neighboring pixels in a frame while temporal redundancy refers to correlation between neighboring frames.
FIGS. 1A-1C illustrate the type of encoded/compressed video frames that are commonly utilized for video standards. FIG. 1A depicts an Intra-frame or I-type frame 200. The I-type frame or picture is a frame of video data that is coded exploiting only the spatial correlation of the pixels within the frame without using information from the past or the future and is utilized as the basis for decoding/decompression of other type frames. FIG. 1B is a representation of a Predictive-frame or P-type frame 210. The P-type frame or picture is a frame that is encoded/compressed using motion compensated prediction from I-type or P-type frames of its past, in this case, I.sub.1200. That is, previous frames are used to encode/compress a present given frame of video data. 205a represents the motion compensated prediction information to create a P-type frame 210. Since in a typical video sequence the adjacent frames in a sequence are highly correlated, higher compression efficiencies are achieved when using P-frames.
FIG. 1C depicts a Bi-directional-frame or B-type frame 220. The B-type frame or picture is a frame that is encoded/compressed using a motion compensated prediction derived from the I-type reference frame (200 in this example) or P-type reference frame in its past and the I-type reference frame or P-type reference frame (210 in this example) in its future or a combination of both. B-type frames are usually inserted between I-type frames or P-type frames. FIG. 2 represents a group of pictures in what is called display order I.sub.1 B.sub.2 B.sub.3 P.sub.4 B.sub.5 P.sub.6. FIG. 2 illustrates the B-type frames inserted between I-type and P-type frames and the direction which motion compensation information flows.
A system for P-frame encoding and decoding is provided and is shown in FIGS. 3 and 4. Referring to FIGS. 3 and 4, a communication system comprising an encoder 300 of FIG. 3 and a decoder 400 of FIG. 4 is operable to communicate a multimedia sequence between a sequence generator and a sequence receiver. Other elements of the video sequence generator and receiver are not shown for the purposes of simplicity. The communication path between sequence generator and receiver may take various forms, including but not limited to a radio-link.
Encoder 300 is shown in FIG. 3 coupled to receive video input on line 301 in the form of a frame to be encoded I(x, y), called the current frame. By (x, y) we denote location of the pixel within the frame. In the encoder the current frame I(x,y) is partitioned into rectangular regions of Mxc3x97N pixels. These blocks are encoded using either only spatial correlation (intra coded blocks) or both spatial and temporal correlation (inter coded blocks). In what follows we concentrate on inter blocks.
Each of inter coded blocks is predicted using motion information from the previously coded and transmitted frame, called reference frame and denoted as R(x,y), which at given instant is available in the frame memory 350 of the encoder 300. The motion information of the block may be represented by two dimensional motion vector (xcex94x(x,y), xcex94y(x,y)) where xcex94x(x,y) is the horizontal and xcex94y(x,y) is the vertical displacement, respectively of the pixel in location (x,y) between the current frame and the reference frame. The motion vectors (xcex94x( ), xcex94y( )) are calculated by the motion estimation and coding block 370. The input to the motion estimation and coding block 370 are current frame and reference frame. The motion information is provided to a Motion Compensated (MC) prediction block 360. The MC prediction block is also coupled to a frame memory 350 to receive the reference frame. In the MC block 360, the motion vectors for each inter block together with the reference frame are used to construct prediction frame P(x, y):
P(x, y)=R(x+xcex94x(x,y), y+xcex94y(x,y)).
Notice that values of the prediction frame are calculated only for inter blocks. For some pixels (x,y) which belong to intra blocks these values will not be calculated. It is also possible to use more than one reference frame. In such case different blocks may use different reference frames.
Subsequently, the prediction error E(x, y), i.e., the difference between the current frame and the prediction frame P(x, y) is calculated by:
E(x, y)=I(x, y)xe2x88x92P(x, y).
In transform block 310, the prediction error for each Kxc3x97L block is represented as weighted sum of a transform basis functions f.sub.ij(x, y),       E    ⁢          (              x        ,        y            )        =            ∑              i        =        1            K        ⁢                  ∑                  j          =          1                L            ⁢                        c          .          s                ⁢                  xe2x80x83                ⁢        u        ⁢                  xe2x80x83                ⁢                  b          .          e                ⁢                  xe2x80x83                ⁢        r        ⁢                  xe2x80x83                ⁢                  r          ⁢                      (                          i              ,              j                        )                          ⁢                  f          .          s                ⁢                  xe2x80x83                ⁢        u        ⁢                  xe2x80x83                ⁢                  b          .          i                ⁢                  xe2x80x83                ⁢                              j            ⁢                          (                              x                ,                y                            )                                .                    
The weights c.sub.err(i,j), corresponding to the basis functions are called prediction error coefficients. Coefficients c.sub.err(i,j) can be calculated by performing so called forward transform. These coefficients are quantized in quantization block 320:
I.sub.err(i, j)=Q(c.sub.err(i,j),QP)
where I.sub.err(i, j) are the quantized coefficients. The operation of quantization introduces loss of informationxe2x80x94the quantized coefficient can be represented with smaller number of bits. The level of compression (loss of information) is controlled by adjusting the value of the quantization parameter (QP).
The quantization block 320 is coupled to both a multiplexer 380 and an inverse quantization block 330 and in turn an inverse transform block 340. Blocks 330 and 340 provide decoded prediction error E.sub.c(x, y) which is added to the MC predicted frame P(x, y) by adder 345. These values can be further normalized and filtered and the result stored in frame memory 350.
Motion vectors and quantized coefficients are encoded using Variable Length Codes (VLC) which further reduce the number of bits needed for their representation. Encoded motion vectors and quantized coefficients as well as other additional information needed to represent each coded frame of the image sequence constitute a bitstream 415 which is transmitted to the decoder 400 of FIG. 4. Bitstream may be multiplexed 380 before transmission.
The special type of the inter coded blocks are copy coded blocks. For copy coded blocks values of both motion vectors and quantized prediction error coefficients I.sub.err are equal to 0.
FIG. 4 shows the decoder 400 of the communication system. Bitstream 415 is received from encoder 300 of FIG. 3. Bitstream 415 is demultiplexed via demultiplexer 410. Dequantized coefficients d.sub.err(i,j) are calculated in the inverse quantization block 420:
d.sub.err(i, j)=Qxe2x88x921(I.sub.err(i, j), QP).
In inverse transform block 430, the dequantized coefficients are used to obtain compressed prediction error by performing inverse transform:             E      .      s        ⁢          xe2x80x83        ⁢    u    ⁢          xe2x80x83        ⁢          b      .              c        ⁢                  (                      x            ,            y                    )                      =            ∑              i        =        1            K        ⁢                  ∑                  j          =          1                L            ⁢                        d          .          s                ⁢                  xe2x80x83                ⁢        u        ⁢                  xe2x80x83                ⁢                  b          .          e                ⁢                  xe2x80x83                ⁢        r        ⁢                  xe2x80x83                ⁢                  r          ⁢                      (                          i              ,              j                        )                          ⁢                  f          .          s                ⁢                  xe2x80x83                ⁢        u        ⁢                  xe2x80x83                ⁢                  b          .          i                ⁢                  xe2x80x83                ⁢                              j            ⁢                          (                              x                ,                y                            )                                .                    
The pixels of the current coded frame are reconstructed by finding the prediction pixels in the reference frame R(x,y) using the received motion vectors and then adding to the compressed prediction error in adder 435:
I.sub.c(x, y)=R(x+xcex94x, y+xcex94,y)+E.sub.c(x, y).
To obtain reconstructed image these values can be further normalized and filtered.
An example of a forward transform is provided by xe2x80x9cH.26L Test Model Long Term Number 6 (TML-6) draft0xe2x80x9d, document VCEG-L45, ITU-T Video Coding Experts Group Meeting, Eibsee, Germany, Jan. 9-12, 2001. The forward transformation of some pixels a, b, c, d into 4 transform coefficients A, B, C, D is defined by:
A=13a+13b+13c+13d
B=17a+7bxe2x88x927cxe2x88x9217d
C=13axe2x88x9213bxe2x88x9213c+13d
D=7axe2x88x9217b+17cxe2x88x927d
The inverse transformation of transform coefficients A, B, C, D into 4 pixels axe2x80x2, bxe2x80x2, cxe2x80x2, dxe2x80x2 is defined by:
axe2x80x2=13A+17B+13C+7D
bxe2x80x2=13A+7Bxe2x88x9213Cxe2x88x9217D
cxe2x80x2=13Axe2x88x927Bxe2x88x9213C+17D
dxe2x80x2=13Axe2x88x9217B+13Cxe2x88x927D
The transform/inverse transform is performed for 4xc3x974 blocks by performing defined above one dimensional transform/inverse transform both vertically and horizontally.
In xe2x80x9cH.26L Test Model Long Term Number 6 (TML-6) draft0xe2x80x9d, document VCEG-L45, ITU-T Video Coding Experts Group Meeting, Eibsee, Germany, Jan. 9-12, 2001 for chroma component, an additional 2xc3x972 transform for the DC coefficients is performed.
DCC(0,0)=(DC0+DC1+DC2+DC3)/2
DCC(1,0)=(DC0xe2x88x92DC1+DC2xe2x88x92DC3)/2
DCC(0,1)=(DC0+DC1xe2x88x92DC2xe2x88x92DC3)/2
xe2x80x83DCC(1,1)=(DC0xe2x88x92DC1xe2x88x92DC2+DC3)/2
Definition of the corresponding inverse transform:
DC0=(DCC(0,0)+DCC(1,0)+DCC(0,1)+DCC(1,1))/2
DC1=(DCC(0,0)xe2x88x92DCC(1,0)+DCC(0,1)xe2x88x92DCC(1,1))/2
DC2=(DCC(0,0)+DCC(1,0)xe2x88x92DCC(0,1)xe2x88x92DCC(1,1))/2
DC3=(DCC(0,0)xe2x88x92DCC(1,0)xe2x88x92DCC(0,1)+DCC(1,1))/2
In xe2x80x9cH.26L Test Model Long Term Number 6 (TML-6) draft0xe2x80x9d, document VCEG-L45, ITU-T Video Coding Experts Group Meeting, Eibsee, Germany, Jan. 9-12, 2001 to obtain values of reconstructed image the results of the inverse transform are normalized by shifting by 20 bits (with rounding).
An example of quantization/dequantization is provided by xe2x80x9cH.26L Test Model Long Term Number 6 (TML-6) draft0xe2x80x9d, document VCEG-L45, ITU-T Video Coding Experts Group Meeting, Eibsee, Germany, Jan. 9-12, 2001. A coefficient c is quantized in the following way:
I=(cxc3x97A(QP)+fxc3x97220)//220
where f may be in the range (0-0.5) and f may have the same sign as c. By //division with truncation is denoted. The dequantized coefficient is calculated as follows:
d=Ixc3x97B(QP)
Values of A(QP) and B(QP) are given below:
A(QP=0, . . . , 31)=[620, 553, 492, 439, 391, 348, 310, 276, 246, 219, 195, 174, 155, 138, 123, 110, 98, 87, 78, 69, 62, 55, 49, 44, 39, 35, 31, 27, 24, 22, 19, 17];
B(QP=0, . . . , 31)=[3881, 4351, 4890, 5481, 6154, 6914, 7761, 8718, 9781, 10987, 12339, 13828, 15523, 17435, 19561, 21873, 24552, 27656, 30847, 34870, 38807, 437 47, 49103, 54683, 61694, 68745, 77615, 89113, 100253, 109366, 126635, 141533];
Video streaming has emerged as one of the essential applications over the fixed internet and-in the near future over 3G multimedia networks. In streaming applications, the server starts streaming the pre-encoded video bitstream to the receiver upon a request from the receiver which plays the stream as it receives with a small delay. The problem with video streaming is that the best-effort nature of today""s networks causes variations of the effective bandwidth available to a user due to the changing network conditions. The server should then scale the bitrate of the compressed video to accommodate these variations. In case of conversational services that are characterized by real-time encoding and point-to-point delivery, this is achieved by adjusting, on the fly, the source encoding parameters, such as quantization parameter or frame rate, based on the network feedback. In typical streaming scenarios when already encoded video bitstream is to be streamed to the client, the above solution can not be applied.
The simplest way of achieving bandwidth scalability in case of pre-encoded sequences is by producing multiple and independent streams of different bandwidth and quality. The server dynamically switches between the streams to accommodate variations of the bandwidth available to the client.
Now assume that we have multiple bitstreams generated independently with different encoding parameters, such as quantization parameter, corresponding to the same video sequence. Since encoding parameters are different for each bitstream, the reconstructed frames of different bitstreams at the same time instant will not be the same. Therefore when switching between bitstreams, i.e., starting to decode a bitstream, at arbitrary locations would lead to visual artifacts due to the mismatch between the reference frames used to obtain predicted frame. Furthermore, the visual artifacts will not only be confined to the switched frame but will further propagate in time due to motion compensated coding.
In the current video encoding standards, perfect (mismatch-free) switching between bitstreams is achieved possible only at the positions where the future frames/regions do not use any information previous to the current switching location, i.e., at I-frames. Furthermore, by placing I-frames at fixed (e.g. 1 sec) intervals, VCR functionalities, such as random access or xe2x80x9cFast Forwardxe2x80x9d and xe2x80x9cFast Backwardxe2x80x9d (increased playback rate) for streaming video content, are achieved. User may skip a portion of video and restart playing at any I-frame location. Similarly, increased playback rate can be achieved by transmitting only I-pictures. The drawback of using I-frames in these applications is that since I-frames are not allowed to utilize temporal redundancy they require much larger number of bits than P-frames.
The above-mentioned references are exemplary only and are not meant to be limiting in respect to the resources and/or technologies available to those skilled in the art.
A new picture or frame type and method of using same is provided. This type of novel frame type is referred to as a SP-picture. SP-picture uses motion compensated predictive coding to exploit temporal redundancy in the sequence. The difference between SP and P-pictures is that using SP-pictures identical frames may be obtained even when different reference frames are used for prediction. This property allows SP-pictures to replace I-pictures in numerous applications such as switching from one bitstream to another, random access, fast-forward, fast-backward. At the same time since SP-frames unlike I-frames are utilizing motion compensated predictive coding they require smaller number of bits than I-frames.
These and other features, aspects, and advantages of embodiments of the present invention will become apparent with reference to the following description in conjunction with the accompanying drawings. It is to be understood, however, that the drawings are designed solely for the purposes of illustration and not as a definition of the limits of the invention, for which reference should be made to the appended claims.